Sonobuoy Conformance Results #204

VestigeJ · 2024-12-20T18:53:56Z

Proposed Changes

Adding the ability to run sonobuoy and get the results from the run of conformance results

the command to run it exists within the makefile and it has been edited to accommodate the long timeout window that conformance results frequently hit within go/ginkgo expectations.

Types of Changes

It can be a quick test by altering the launchSonobuoyTests("quick") but the default is to run the full conformance suite from upstream. Primarily to be used during patch validation runs but it could be used more frequently in other ways to quickly check a clusters status. Although if you have a cluster running the fastest and easiest way to run conformance is to do it the simple way upstream does it with a simple yaml deployment.

---
apiVersion: v1
kind: Namespace
metadata:
  name: conformance
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    component: conformance
  name: conformance-serviceaccount
  namespace: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    component: conformance
  name: conformance-serviceaccount-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: conformance-serviceaccount
subjects:
- kind: ServiceAccount
  name: conformance-serviceaccount
  namespace: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    component: conformance
  name: conformance-serviceaccount
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '/metrics'
  - '/logs'
  - '/logs/*'
  verbs:
  - 'get'
---
apiVersion: v1
kind: Pod
metadata:
  name: e2e-conformance-test
  namespace: conformance
spec:
  containers:
  - name: conformance-container
    image: registry.k8s.io/conformance-amd64:v1.31.2 #EDIT THIS FOR YOUR CLUSTER
    imagePullPolicy: IfNotPresent
    env:
    - name: E2E_FOCUS
      value: "\\[Conformance\\]"
    - name: E2E_SKIP
      value: ""
    - name: E2E_PROVIDER
      value: "skeleton"
    - name: E2E_PARALLEL
      value: "false"
    - name: E2E_VERBOSITY
      value: "4"
    volumeMounts:
    - name: output-volume
      mountPath: /tmp/results
  volumes:
  - name: output-volume
    hostPath:
      path: /tmp/results
  restartPolicy: Never
  serviceAccountName: conformance-serviceaccount

Testing

Checklist:

If your PR changes anything on or related to Jenkins, run it pointing to your branch to make sure it's okay.
not yet
Verify code lint; we should not have errors.
Update the documentation if needed.
✅
Update makefile and docker run if adding new tests.
✅
Run your tests at least 4 times with all configurations needed and possible.
✅
If needed test with different os types.
✅

Linked Issues

Further Comments

pkg/testcase/conformance.go

config/optional_files/get_sonobuoy.sh

pkg/testcase/conformance.go

VestigeJ · 2024-12-20T19:57:24Z

I'll be running golangci-lint run a lot more before creating PRs in the future.... Apologies

entrypoint/sonobuoyconformance/sonobuoyconformance_suite_test.go

entrypoint/sonobuoyconformance/my-sonobuoy-plugins

pkg/testcase/conformance.go

pkg/testcase/networkconnectivity.go

mdrahman-suse · 2024-12-20T21:43:20Z

pkg/testcase/networkconnectivity.go

+			cluster.Config.Product) + " https://127.0.0.1:6443/healthz",
+	}
+
+	servers, err := shared.GetNodesByRoles("control-plane")


nit: servers -> cpNodes

controlPlaneNodes is my replacement string again long for comprehension considering it shouldn't be a string that is typed often but that is read more than re-typed.

pkg/testcase/conformance.go

pkg/testcase/networkconnectivity.go

scripts/install_sonobuoy.sh

shared/cluster.go

pkg/testcase/conformance.go

mdrahman-suse · 2024-12-20T21:59:06Z

I'll be running golangci-lint run a lot more before creating PRs in the future.... Apologies

No worries, you can just run make pre-commit before pushing the changes

pkg/testcase/conformance.go

ShylajaDevadiga · 2025-01-06T21:27:49Z

pkg/testcase/cluster.go

-	sonobuoyVersion := customflag.ServiceFlag.External.SonobuoyVersion
-	err := shared.SonobuoyMixedOS("install", sonobuoyVersion)
+// TestDisplayClusterDetails used to display cluster details.
+func TestDisplayClusterDetails() {


Are we not using this here? https://github.com/rancher/distros-test-framework/blob/main/pkg/template/helper.go#L79
Update path is all tests thats using this method

This method is still exposed publicly from the conformance file

pkg/testcase/networkconnectivity.go

.gitignore

pkg/testcase/conformance.go

fmoral2 · 2025-01-17T12:05:02Z

shared/cluster.go

+	fmt.Println("serverVersion: ", strings.Split(serverVersion, "\n"))
+	fmt.Println("re:", re)
+	match := re.FindStringSubmatch(serverVersion)
+	fmt.Println("match: ", match)


if you want to print use logs please in all those println

shared/cluster.go

entrypoint/conformance/sonobuoy_test.go

entrypoint/conformance/conformance_suite_test.go

pkg/testcase/conformance.go

mdrahman-suse · 2025-02-10T13:29:23Z

pkg/testcase/conformance.go

+	if deleteWorkload {
+		cmd = "sonobuoy delete --all --wait --kubeconfig=" + shared.KubeConfigFile
+		_, err = shared.RunCommandHost(cmd)
+		Expect(err).NotTo(HaveOccurred(), "failed cmd: "+cmd)
+		err = shared.InstallSonobuoy("delete")
+		if err != nil {
+			GinkgoT().Errorf("error: %v", err)
+			return
+		}
+	}


Can probably re-use cleanupTests func here

Initially some months ago I specifically didn't want to change the existing mixedOS test functionality, but good call out now.

mdrahman-suse · 2025-02-10T13:30:44Z

pkg/testcase/conformance.go

+	cmd = "sonobuoy retrieve --kubeconfig=" + shared.KubeConfigFile
+	testResultTar, err := shared.RunCommandHost(cmd)
+	Expect(err).NotTo(HaveOccurred(), "failed cmd: "+cmd)
+
+	cmd = "sonobuoy results  " + testResultTar
+	res, err = shared.RunCommandHost(cmd)
+	Expect(err).NotTo(HaveOccurred(), "failed cmd: "+cmd)
+	Expect(res).Should(ContainSubstring("Plugin: mixed-workload-e2e\nStatus: passed\n"))


Can probably re-use getResults and parseResults func here

mdrahman-suse

Added some comments, else LGTM... also needs rebasing

fmoral2 · 2025-02-26T18:31:51Z

entrypoint/conformance/conformance_suite_test.go

+	}
+
+	os.Exit(m.Run())
+	os.Exit(1)


this code line is unreachable